<html>
<head><meta charset="utf-8"><title>Handling streaming data efficiently · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html">Handling streaming data efficiently</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="215265304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215265304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215265304">(Nov 01 2020 at 22:19)</a>:</h4>
<p>I'm looking for an efficient pattern to handle streaming data. I'd like the writer to never block, the data to be streamed into a temporary file, and any reader to be able to catch up and then block on new data without busylooping.</p>



<a name="215265321"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215265321" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215265321">(Nov 01 2020 at 22:19)</a>:</h4>
<p>There's no simple way for a reader to just block on new data appearing in a temporary file.</p>



<a name="215265330"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215265330" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215265330">(Nov 01 2020 at 22:20)</a>:</h4>
<p>(And inotify/fanotify seems like massive overkill here, though if I have to go that route I can.)</p>



<a name="215265407"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215265407" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215265407">(Nov 01 2020 at 22:22)</a>:</h4>
<p>I also want to avoid having the reader just sleep and then re-check.</p>



<a name="215266377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215266377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215266377">(Nov 01 2020 at 22:45)</a>:</h4>
<p>The writer is obtaining data from the network, and writing it to a temporary file. There's some additional work to be done when the data is complete and the network connection is closed. Zero or more readers want to be able to follow the data as it's obtained, but want to be able to efficiently block until there's more data to read.</p>



<a name="215285037"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215285037" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215285037">(Nov 02 2020 at 07:12)</a>:</h4>
<p>Can you use <del>a shared memory map containing the length and</del> a cross-process semaphore / condition variable for notifications?</p>



<a name="215285095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215285095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215285095">(Nov 02 2020 at 07:13)</a>:</h4>
<p>The writer appends to file, <del>updates length in the shm,</del> then pings the readers. Actually, you don't even need the shm part, the readers can wait, keep reading until the end, then go back to sleep</p>



<a name="215348141"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348141" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348141">(Nov 02 2020 at 17:07)</a>:</h4>
<p>This is within the same process (at least for now).</p>



<a name="215348166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348166">(Nov 02 2020 at 17:08)</a>:</h4>
<p>But I don't want to buffer in memory, the data may be large.</p>



<a name="215348305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348305">(Nov 02 2020 at 17:08)</a>:</h4>
<p>I think the approach above works even better in-process (since IPC cvars are harder to come around). And there's no buffering needed.</p>



<a name="215348417"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348417" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348417">(Nov 02 2020 at 17:09)</a>:</h4>
<p>A condition variable like that did occur to me, but it would require care to avoid racing, and also afaict Rust's condition variables <em>require</em> using a mutex.</p>



<a name="215348556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348556">(Nov 02 2020 at 17:10)</a>:</h4>
<p>Hence trying to find an existing structure rather than building one, though I can do that if there isn't one.</p>



<a name="215348620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348620">(Nov 02 2020 at 17:11)</a>:</h4>
<p>Most need one, you have to acquire a mutex, then sleeping on them will release it</p>



<a name="215348627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348627">(Nov 02 2020 at 17:11)</a>:</h4>
<p><span class="user-mention" data-user-id="203546">@Laurențiu Nicola</span> What did you have in mind to handle the "wait/read/sleep" pattern safely?</p>



<a name="215348688"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348688" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348688">(Nov 02 2020 at 17:11)</a>:</h4>
<p>Well, the writer does:</p>
<ul>
<li>wait for data from network</li>
<li>write to file</li>
<li>signal cvar</li>
<li>go to step one</li>
</ul>
<p>The readers open the file  (one handle per reader, this needs to happen before the loop), then they:</p>
<ul>
<li>sleep on the cvar</li>
<li>read from the file</li>
<li>process data</li>
<li>read again, until there's nothing more to read</li>
<li>go to step one</li>
</ul>



<a name="215348913"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215348913" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215348913">(Nov 02 2020 at 17:12)</a>:</h4>
<p>(If it helps, there is only one writer here.)</p>



<a name="215349263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215349263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215349263">(Nov 02 2020 at 17:15)</a>:</h4>
<p>There's some inefficiency (each reader will read the same data from disk), but in practice it should be cached</p>



<a name="215349297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215349297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215349297">(Nov 02 2020 at 17:15)</a>:</h4>
<p>That pattern would introduce a race condition. The reader reads from the file, processes the data, runs out of data to read, then the condition variable is signaled before the reader goes to sleep, then the reader goes to sleep and never gets woken up.</p>



<a name="215349506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215349506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215349506">(Nov 02 2020 at 17:16)</a>:</h4>
<p>(Not worried about repeatedly reading the same data from disk. In practice there will commonly be either zero or one readers.)</p>



<a name="215350049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350049">(Nov 02 2020 at 17:20)</a>:</h4>
<p>Ouch, right. My feeling is that it can work, though. The problem is that you're not holding the cvar lock while reading (which isn't desirable), so you could check if more data was written before sleeping on the cvar.</p>



<a name="215350157"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350157" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350157">(Nov 02 2020 at 17:21)</a>:</h4>
<p>Say the writer publishes a version number (number of <code>write()</code> calls), and the readers check it before sleeping.. but it's still a bit awkward</p>



<a name="215350225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350225">(Nov 02 2020 at 17:21)</a>:</h4>
<p>Still needs an atomic-check-and-sleep primitive.</p>



<a name="215350356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350356">(Nov 02 2020 at 17:22)</a>:</h4>
<p>It's atomic, because you acquire the mutex, do the version check, then sleep</p>



<a name="215350395"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350395" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350395">(Nov 02 2020 at 17:23)</a>:</h4>
<p>The writer has to acquire the mutex, increase the version, then signal the cvar</p>



<a name="215350398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350398">(Nov 02 2020 at 17:23)</a>:</h4>
<p>The writer shouldn't have to acquire a mutex and wait on readers though.</p>



<a name="215350439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350439">(Nov 02 2020 at 17:23)</a>:</h4>
<p>Yeah, that's not great. At least they're short-lived locks</p>



<a name="215350463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350463">(Nov 02 2020 at 17:23)</a>:</h4>
<p>What's annoying is that I know Linux futexes support the exact operation that I want...</p>



<a name="215350534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350534">(Nov 02 2020 at 17:24)</a>:</h4>
<p>Ah, I've never looked into those</p>



<a name="215350581"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/Handling%20streaming%20data%20efficiently/near/215350581" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/Handling.20streaming.20data.20efficiently.html#215350581">(Nov 02 2020 at 17:24)</a>:</h4>
<p>They're the underlying mechanism used to implement any kind of sleeping lock on Linux.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>